Skip to content

Add new EF 11 features - #296

Open
ChrisJollyAU wants to merge 75 commits into
CirrusRedOrg:masterfrom
ChrisJollyAU:ef11-features-1
Open

Add new EF 11 features#296
ChrisJollyAU wants to merge 75 commits into
CirrusRedOrg:masterfrom
ChrisJollyAU:ef11-features-1

Conversation

@ChrisJollyAU

Copy link
Copy Markdown
Member

New

Both: Add new Parse translator
Both: Implemented a jet compatible NULLIF to IIF... sql translation

LibRed only
Full BMP plane coverage of General and General Legacy collating orders
Implemented tailorings for all 30 non CJK collating orders
Database can be created with a specific collating order
Implemented INSERT INTO ... SELECT
Implemented SELECT ... INTO
Implemented NULLIF function
Implemented FULL JOIN

Bug Fixes

Continue the Autonumber counter past the int32 wrap
Fix the migrations-lock hang in JetHistoryRepository
Take the relocation pointer from the slot's first 4 bytes, not its width

ChrisJollyAU and others added 30 commits August 22, 2026 01:00
Probed ACE (OLE DB 16.0/12.0) at the counter boundary: there is no overflow
error. The TDEF high-water (0x14) is a plain signed int32 and the next id is
0x14 + increment computed unchecked, so an ascending counter runs
... 2147483647, -2147483648, -2147483647 ... and a descending one mirrors it.
ACE writes the wrapped id to 0x14 and carries on. A wrapped id that is already
occupied is an ordinary duplicate-key rejection, and ACE burns the id anyway
(0x14 advances despite the failed insert) so the next insert steps over it.

LibRed generated the same wrapped id but then wedged: UpdateTdefCounters'
monotone guard - the deliberate KB 884185 immunity - read the wrapped value as
going backwards and left 0x14 pinned at int.MaxValue, so every later auto insert
reissued int.MinValue. The damage was the on-disk 0x14, so ACE opening the file
failed the same way.

AssignAutoNumbers now returns a per-value flag array of the ids it generated,
and UpdateTdefCounters takes a generated id as the new high-water
unconditionally: it came from 0x14 + increment, so it is the next value in the
sequence by construction, wrap included. The monotone guard now applies only to
caller-supplied explicit ids, which is the only case KB 884185 was ever about.
Confirmed that guard is still needed: making explicit ids leave 0x14 untouched
instead fails 5 tests, including ACE reusing id 1 after LibRed bulk-writes rows
1-3 - explicit ids are how data gets into a counter column, and 0x14 is the only
record of where the counter is.

Also moves the AssignAutoNumbers doc comment onto the method (it had been
orphaned above MaterializeLongValues).

Spec: page-02a-tdef.md gains a verified wrap note under 3.1, and the 0x14/0x18
rows plus the appendix entry record that the value wraps.

Tests: AceAutoNumberOverflowProbeTest covers both engines at both boundaries,
the explicit-int.MaxValue route to the wrap, and the occupied-wrapped-id case.
LibRed.Core 481/481, LibRed.Engine 898/898, LibRed.Ado 47/47.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Implemented JetParseTranslator to map .NET Parse methods (e.g., int.Parse, double.Parse) to Jet/Access SQL type conversion functions (CBYTE, CDBL, CINT, CLNG, CDEC). Registered the translator in JetMethodCallTranslatorProvider. Updated related tests to expect Jet/Access SQL syntax for Parse operations instead of SQL Server-style CAST/CONVERT.
Squashes a batch of engine fixes and the test work that found them.

Engine and format fixes (LibRed.Core / LibRed.Engine / LibRed.Ado):

- Empty Binary index key is the start flag alone (7F asc / 80 desc), not a
  zero-padded chunk. ACE oracle: ACE writes the index, LibRed reads the stored
  entry back and re-encodes it. A LibRed-written empty key previously could not
  compare equal to an ACE-written one in the same index.
- Office-Standard EncryptionInfo is parsed from the declared frame
  (len@0x299, blob at 0x29B) instead of scanning page 0 for a signature. The
  length is authoritative for ACE itself - a file with key + descriptor but
  len = 0 is read by Access as plaintext - so the old scan opened databases ACE
  cannot. Verified by falsification: with the scan restored, LibRed opens such a
  file successfully.
- UPDATE (Flag 4) and DELETE (Flag 5) stored action queries are recognised and
  reported as not-yet-executable, rather than falling into the generic
  unsupported bucket. Flags verified against ACE-authored procedures.
- Row pointers are bounds-checked before decode, so a corrupt index entry raises
  InvalidDataException instead of decoding arbitrary bytes.
- Commit validates every overlay page against the committed image it was derived
  from, under a per-file publish gate, so overlapping writers conflict
  deterministically instead of losing an update. A failed publication restores
  the already-published prefix and keeps the transaction rollbackable, reporting
  both the publish failure and any restore failure.
- Schema-changing commits advance a per-file catalog generation; other open
  connections reload their parsed catalog on next access, while plain DML does
  not force a reload.
- Function argument arity is validated against ACE's JES, including Jet quirks
  (two-argument IIf yields Null). Aggregates go through the same contract.
- SQL COMMIT/ROLLBACK reconciles the ADO transaction handle.

Page scope is now reader/writer rather than a mutex. A statement that cannot
write (SELECT, set operation, system-variable select) takes it shared, so
concurrent readers on one file still run together; everything else takes it
exclusive for the whole statement, which is what makes a multi-page write atomic
to readers. Anything not provably read-only takes the exclusive scope - the
shared scope cannot be upgraded and says so rather than deadlocking. Parsing
happens before the scope is taken.

Test suite:

- Wall-clock guards replaced with structural assertions (the planner is asked
  directly whether the rewrite engaged), and ThrowsAny<Exception> replaced with
  specific exception types plus message assertions.
- Shared AceTestDatabase / TemporaryDatabase helpers, in test/LibRed.Shared so
  the EF functional projects (which glob test/Shared wholesale and do not all
  reference LibRed.Core) are unaffected - EFCore.Jet.FunctionalTests had stopped
  building.
- Temp databases are released per test. A database opened and abandoned by a
  static Fresh() helper kept its file locked for the whole process, so the copy
  survived every cleanup path; ~22 GB had accumulated in %TEMP%. Handles are now
  owned by the helper, closed before deletion, and released when the test ends,
  with a process-exit backstop. Peak temp copies during an engine run: 609 -> 19.
- The five ACE-driving classes share one xunit collection. Concurrent ACE use
  faults natively (SEHException, then 0xC0000005 kills the run): those classes
  alone crashed 3 of 3 back-to-back runs, the other ~950 tests were clean 3 of 3.
  Parallelism is not disabled - only those five are serialized, against each
  other.
- New: SchemaVisibilityTests (cross-connection catalog freshness, previously
  untested), reader/writer scope tests, and a zero-length-descriptor guard. Each
  was verified to fail with its mechanism disabled.
- The Access-output legacy password comparison is restored as a fixture-gated
  test that skips with a reason (LIBRED_ENCTEST_DIR) instead of silently passing.

Docs updated: transactions.md (scope semantics), page-00-database.md (descriptor
framing, password fixtures), page-03-04-index-btree.md (empty Binary key),
system-catalog.md (action flags), functions.md (arity).

LibRed.Core 599, LibRed.Engine 962, LibRed.Ado 47 - all passing, no crashes,
zero temp files leaked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…m tables

Access 2010 added a second "General" sort order, and the collation version byte
selects a genuinely different weight table rather than being metadata: the same
values indexed in a v0 and a v1 database differ in all 28 samples.

v1's primaries are the Windows NLS (Script Member, Alphabetic Weight) pair
verbatim - "apple" is 0E02 0E7E 0E7E 0E48 0E21 - where General Legacy compacts
the same ordering into one byte per character. That is why v1 can be derived
from a published table while v0 has to be measured.

Also records what ACE needs MSysComplexColumns for, which is its own feature
rather than part of the collation work.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…gaps are for

Bytes 0x0B-0x0E (page-0 0x6E-0x71) are ONE Windows LCID with the sort-order
version in its otherwise-unused top byte: LANGID at 0x0B, sort id at 0x0D,
version at 0x0E. The sort id is what separates a Windows alternate sort order
from its base locale - German Phone Book 0x00010407 against German 0x00000407 -
and they differ in nothing else. The spec had warned for months that 0x0D was
"0 in every file seen, keep an eye on it".

The gaps in the v0 letter table are insertion slots for language letters. It
steps by +2 everywhere except B-C, Q-R and X-Y, and Spanish lands on exactly the
free value in each relevant gap. The second byte is a SUB-POSITION ordering
letters that share a slot, proved by locales that put several in one.

Version 1 is not General-only: Croatian and Romanian ship in both generations.
Five of DAO's collating orders are dead metadata, byte-identical to General, so
appearing in the UI does not imply an implementation. DAO can author a locale
but not a sort-order version, which is what fixtures need Access itself for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JetLocaleTailoring keys per-locale overrides by STRING rather than character, so
a contraction works: a digraph weighing as one letter is the inverse of the
existing expansions, and the one primitive neither encoder had. Matching is
greedy longest-first with no backtracking, and looks up the ORIGINAL text before
the uppercased text, which is what lets Turkish disagree with invariant casing.
Only Hungarian doubles, and that test must run before the longest match.

Tailoring is not only insertion. Six devices, all inside the existing framing:
insertion, contraction, expansion, secondary retune, remapping the base table,
and reordering. Empty tailorings mean "measured to need no change", which is
different from having none.

Also fills in the General diacritic table and fixes the long s, which is a
letter of its own rather than a fold onto s - found by testing each locale
against a set far wider than its own tailoring.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…venance

v0 IS the NT4-era NLS order, renumbered into one byte, and the compaction is
ORDER-PRESERVING. Sorting every character by the Windows NT 4.0 - Server 2003
table's (SM, AW) primary and checking v0's bytes come out non-decreasing keeps
507 of 510 strictly-ordered pairs and 947 of 955 ties, with 12 of 14 blocks
perfect. So the +2 stride, the language-letter insertion gaps and the 0x79 page
are one decision rather than three observations. Jet also NARROWED it: 88 of the
552 v0 ignorables are weighted by NLS and dropped anyway, an editorial call no
published table would reveal.

Locales SHARE the block tables, with per-locale deltas in their own tailoring,
which is what makes 21 orders cost 27 entries between them.

A ligature weighs as its DECOMPOSITION - there is no ligature mechanism in the
format. Components are weighed individually and never re-enter the contraction
matcher, and decomposition sits below the tailoring because some locales do not
decompose at all.

The last gap was the word-sort ignorables, 20 of them rather than 3: every dash,
the Arabic harakat, and fullwidth apostrophe and hyphen, which share their ASCII
counterparts codes exactly - the one place fullwidth really does collapse onto
ASCII, unlike the letters. Coverage is 2147/2147 for all 23 orders.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Prefix compression covers the WHOLE entry including its trailer, not just the
key bytes. Reading it as key-only worked until a page held many equal keys,
which is exactly what a full-BMP sweep produces - thousands of ignorable
characters all encoding to the same empty key - and then the reader rejected the
page outright.

Found by the probes added here rather than by a test written for it, which is
the argument for sweeping a whole range instead of sampling it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every character ACE stores a key for, LibRed now encodes identically - 63,422 of
them. No published table describes v0, since its primaries are a Jet compaction
rather than the NLS weights, so ACE ITSELF is the source:
SortKeyTableV0GeneratorTest inserts every code point into an indexed text
column, reads the stored keys back and writes the embedded resource (74 KB;
63,105 weights, 40 word-sort ignorables, 276 kana). Far past anything
hand-maintainable, and hand-transcribing hex is exactly the work that introduces
a wrong byte nobody notices.

Two things only a full sweep shows: ACE weighs every CJK ideograph and the
entire private-use area, and across all 65,536 code points it refused exactly
one.

KANA take a two-byte primary 7F <sound>, with voicing as an ordinary secondary
and the small/normal distinction bit-packed into a section of its own - three
per byte, two bits each, most significant first, under a 10 marker. Two rules
only multi-character strings reveal: the halfwidth voicing marks are COMBINING
(alone they look ignorable, which is what hid it), and the inline section's
introducer becomes FF 01 when a kana section precedes it.

The prolonged sound mark lengthens the preceding kana's VOWEL, which is what the
character means - がー is "ga" lengthened by "a", not by "ga" - so the vowel is a
property of each kana and has to be measured per character rather than derived.

Also: inline positions count primary WEIGHTS, not bytes, and the hand-verified
expansions stay ahead of the measured table because a key cannot show whether
two bytes are one weight or two.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
v1 now encodes all 63,422 BMP characters exactly as ACE stores them, matching
v0. It was 29,776 differences when the sweep started.

Most of it was four structural rules rather than missing weights, which is why
the sweep was worth running before generating anything: a Han character takes a
FOUR-byte primary (the marker FD FF then its own weights) and no secondary,
which alone was 28,200 wrong keys; Hangul jamo take (AW, DW) as the primary with
no marker; a zero alphabetic weight means NO primary at all; and where something
precedes it, such a weight FOLDS into the one before rather than taking a slot.

The rest is that the published Server 2008 table is not quite what ACE carries.
That identification came from 25 reconstructed keys, all Latin and symbols, and
it holds for 57,793 characters and fails for 501 - Balinese and Canadian
syllabics get Latin weights, and the Arabic harakat and several ligature blocks
differ. Scripts added or reweighted since. Rather than hunt for the right NLS
revision, the disagreements are measured and embedded (2.0 KB), along with 5,082
characters ACE treats as wholly ignorable that the published file has no entry
for at all.

An override stores raw primary and secondary bytes, not (SM, AW, DW) weights:
that reading assumes a two-byte primary carrying one secondary, and ACE breaks
it both ways - the harakat have a secondary and no primary, the Lao vowels take
a one-byte primary. A primary byte can also BE 0x01, so the section delimiter is
the last 0x01 in a key rather than the first. Splitting at the first made five
characters look like an unknown mechanism; measuring them in combination showed
ordinary two-weight expansions.

Kana turn out to be shared: same sound weights, same section, byte for byte
under both orders, so JetKanaSection is extracted rather than duplicated. Two
narrow differences remain - v1 weighs a compatibility form by its base kana's
sound where v0 gives it its own, and five kana absent from the v0 table come
from v1's own script member 3, whose smallness cannot be inferred from reaching
that path.

A generator must run with its own resource suppressed. It records where the
encoder DISAGREES with ACE, so measuring an encoder that already consults it
would find no disagreements and write an empty file.

Behaviour change: an unassigned character such as U+0378 no longer throws. ACE
stores an empty key for it, so refusing would reject a value the engine accepts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ACE stores an index entry of at most 510 bytes as built. At exactly 510 it comes
back byte-for-byte; a value needing 511 comes back as 510 with the weights cut
short and the last two bytes replaced by a value that varies with the string - a
truncated key plus a checksum, which is why two long values never collide. The
checksum function is not known, so LibRed cannot reproduce a truncated key and
refuses the value instead. It was writing the full-length key, which put bytes
in the index ACE would never write, and a wrong index key is silent: ACE writes
its own into the same index and a seek misses rows.

The cap is on the whole ENTRY, not per column. Two 200-character text columns
weigh about 404 bytes of key each - comfortably under the cap individually - and
ACE stores their combined entry hashed at 510. A per-column check would have let
that through.

Because it limits weights and not characters, what it buys varies with collation
and script, which is the practical cost of General over General Legacy and is
invisible in the schema: 255 characters for v0 Latin (the column limit is
reached first), 253 for v1 Latin, 169 for v1 accented, 127 for v1 Han.

Above the BMP the two orders disagree completely, measured over all of planes 1
and 2 and sampled across all sixteen. v0 ignores astral characters entirely -
every one gets the empty key - so under General Legacy an astral character is
invisible to the index. v1 weighs BOTH surrogate halves, each looked up like any
other character: U+10000 is 7F B002 B4F8 01 3F 3F 00.

Only the high surrogates to U+D87F carry weights, so from plane 3 upward the
high half is ignorable and the low one stands alone, and those planes collapse
onto 1,024 keys. The fix v1 needed was therefore narrow: an unweighted surrogate
is ignorable rather than an error. The tempting reading of the plane-3 samples -
"the high surrogate contributes nothing" - is wrong, and skipping every high
surrogate breaks all 131,068 characters of planes 1 and 2.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… plus a byte

An apostrophe or hyphen carries no primary weight and instead appends a record
holding the position it sat at. That position is SIXTEEN bits, big-endian, with
bit 15 set - [MS-UCODEREF] gives SpecialWeightType as (Position: 16 bit integer,
ScriptMember, PrimaryWeight), emitted as "Byte1 = Position >> 8, Byte2 =
Position & 0xff". The 0x80 is not a marker byte at all.

LibRed read it as a marker followed by one position byte and truncated the rest.
The two readings agree below 0x100 and diverge above it, and the offset
0x07 + 4 x position passes 0xFF at position 62 - so a hyphen at character 63 is
81 03 where LibRed wrote 80 03, and at 250 it is 83 EF where LibRed wrote 80 EF.

Every indexed value with an apostrophe or hyphen past character 62 therefore got
a wrong key. A hyphenated name in a 255-character column is enough, and nothing
caught it: single characters encode correctly, short strings encode correctly,
and the field only overflows when a value is long enough. It surfaced while
reverse-engineering something unrelated. The lesson is in the spec beside it -
measure combinations, not only characters, because a per-character sweep can be
exhaustive over all 63,422 and still miss a whole class of bug.

Measured against ACE across positions 10 to 250 under both sort orders.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…values

Past 510 bytes ACE keeps the first 508 and replaces the rest with two bytes
derived from what it dropped, which is why two long values sharing a prefix
still sort apart. LibRed refused those values because the function was unknown.
It is known now, and they are truncated exactly as ACE truncates them.

Recovered by measurement rather than found documented. Three tails differing in
one byte show the function is affine over GF(2) - L(0xA3) ^ L(0x13) = L(0xB0)
exactly - and 173 observations show it is shift-invariant, so a byte at distance
d from the end contributes S^(d-1) of itself whatever the length.

Sweeping all 65,536 polynomials in five framings found NOTHING, and that
negative was the clue. The standard reflected update is
crc = (crc >> 8) ^ T[(crc ^ b) & 0xFF], passing the byte through the table;
ACE computes crc = (crc >> 8) ^ T[crc & 0xFF] ^ b and injects it raw. Wrong
injection point, so no polynomial could ever have matched. The step operator
then came out of Gaussian elimination over the measured contributions and
predicts all 657 of them - no name for the algorithm required. There is no
initial value and no final XOR.

The limit is on the whole ENTRY, not per column, which the measurement also
settled: two 200-character text columns weigh about 404 bytes of key each,
comfortably under the cap individually, and ACE stores their combined entry
truncated. A per-column check would have let that through.

Still refused where the dropped bytes hold an inline word-sort record. That
cannot be verified even in principle - the record sits in the part ACE
discarded, so what it contained is unobservable, and if ACE recomputes its
position when truncating then the checksum's input is not what is reconstructed
here. Guessing would write a silently wrong key.

Text columns now index to the full 255 characters again rather than being
refused past 127 to 253, depending on collation and script.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
French tailors no letter at all. It is General with the diacritic section
written BACKWARDS, so accents are weighed from the end of the word and
cote < cote-with-acute-on-e ... in short, coté sorts before côte where General
has it the other way round. [MS-UCODEREF] names the flag IsReverseDW and gives
both halves of the rule: the run of default diacritics comes off the LEFT rather
than the right, and what remains is written right to left.

Verified against ACE byte for byte. côté is [02 12 02 0E], trimmed to [12 02 0E]
and stored as 0E 02 12. Across all of Latin-1 and Latin Extended-A with accents
doubled and tripled per string, 1,289 values, zero differences.

It had been recorded as "unclassified, secondary-section tailoring", which
described the symptom rather than the rule, and the reason is worth keeping: a
word with ONE accent encodes identically under both orders, and the sample set
that measured every locale against General contained no two-accent word. The
rule was invisible to the measurement rather than absent from it - the same
blind spot as the inline position field.

LibRed can now also CREATE a French database, which follows for free: creating
one requires encoding the order, because the system-table indexes are built on
the way. That circularity is why measuring French needed DAO to author the file
first. ACE indexes into a LibRed-created French database with every key
identical to LibRed's own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The script members were derived by measuring ACE, one class at a time, and the
GetWindowsSortKey pseudocode names every one of them: PUNCTUATION 6 is the
word-sort class, JAMO_SPECIAL 4 the Hangul jamo, EXTENSION_A 5 the Han shape,
NONSPACE_MARK 1 the "no primary, only a secondary" rule. Everything at or below
MAX_SPECIAL_CASE goes to its SpecialCaseHandler - which is exactly the set of
classes that needed bespoke handling here.

Script member 3 is EASTASIA_SPECIAL, not "kana", so the constant is renamed.
That also turns an unexplained list into one rule: the class reserves
PW_REPEAT 0 and PW_CHO_ON 1, and the seven characters ACE gives the unweighted
FF FF primary are exactly those - the iteration marks and the lone prolonged
sound mark.

The 01 01 01 before a word-sort record is not an introducer but three SECTION
SEPARATORS. The frame is primaries 01 diacritics 01 case 01 extra 01 specials
00, and Access emits it with the case-weight section EMPTY - which is the
mechanism behind case and width folding, since width is bit 0 of the Case
Weight. MIN_DW = 2 is the default secondary whose trailing run gets trimmed.

Three further notes recorded in the spec. The contraction limit corroborates
v0's provenance independently - 2 and 3 characters on NT4 through Server 2003,
4 to 8 from Vista, and every v0 tailoring here tops out at three, which is the
same generation the weight-table comparison identified by a different route.
The FD FF Han primary is NOT the Windows 7 three-byte weight, which is three
bytes and postdates the table Access froze. And Access PACKS the East Asia
extra weights three flags to a byte where Windows uses one byte per character.

Nothing in that source covers the 510-byte cap, truncation or the checksum: a
useful negative, since it means those are Jet inventions that had to be measured
rather than looked up.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These three were refused on the grounds that "the v1 encoder has no tailoring
hook: its primaries are 2-byte NLS values, a different shape". That was a fact
about the encoder, not about the orders. Measured, each is General v1 plus a
small override set using the same six devices every version-0 locale uses, and
all three are byte-identical to each other - the same order under three LCIDs,
so one table serves them.

The letters land where the Croatian alphabet puts them: L 0E48, LJ 0E4A, M 0E51;
D 0E1A, DZ-digraph 0E1D, D-with-stroke 0E1E. So the three digraphs are
contractions, which is why the loop had to become indexed. TailoredWeight
already carried byte[] primaries, so a two-byte primary needed no change at all.

Two things the conformance range had to be WIDENED to find, and both would have
written silently wrong keys for ordinary Croatian text:

  - The caron retune reaches further than a hand-picked list of letters showed.
    Eight more, and one of them moves its PRIMARY rather than only its accent.
  - Expansion components were going straight to the base table, so the
    precomposed digraph U+01C4 encoded as D + Z-with-caron instead of D + the
    tailored Z-with-caron. Components take the LOCALE's letters - the rule
    version 0 already followed - but must not re-enter the contraction matcher,
    or expanding a ligature could trip a digraph the original text never had.

Version-1 fixtures were asserted over 447 values where version-0 ones got 2,444,
because the extended blocks were once measured for v0 only. That is no longer
true, and the narrowing only hid ground: it is removed, and all 27 fixtures now
run the same 2,444 values with zero mismatches.

One genuine ACE asymmetry recorded: U+016C takes the retuned secondary while
lowercase U+016D keeps General's, identically in all three locales. Every other
letter is case-symmetric, all three digraphs included.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…n 0)

General v0 gave a secondary-only character a slot of its own in the secondary
section. ACE folds it into the preceding weight instead, adding its value:
Thai ไก่ is two weights with secondaries 03 06 - the tone mark's 03 added to the
consonant's 03 - not three weights. Emitting a slot desynchronises the whole
section from the primaries, so everything after the mark is wrong too.

This affects ALL 28 orders, not one. The class is "secondary-only" characters,
and it holds Thai tone marks, every Hebrew niqqud, the Cyrillic combining marks
and three Greek ones - any indexed text with a combining mark following a base
character.

The version-1 encoder has had this rule since the full-BMP work; version 0 never
got it. It surfaced only when the Thai block entered the conformance range,
because the rule needs a mark AFTER a base character and a per-character sweep
cannot place one there. Nor could comparing one locale against another: both
were wrong identically, so only comparing against ACE shows it.

The conformance range gains the Thai block and words, which is what caught it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Thai writes the five vowels e/ae/o/ai/ai-maimuan BEFORE the consonant they are
pronounced after, and collation follows speech. That was recorded as needing
REORDERING, a device nothing else here uses, and it is why Thai stayed
unimplemented as the last non-CJK order.

Measurement says it is an ordinary contraction - the same device Croatian's lj
uses. ACE gives the pair a SINGLE weight at the consonant's own primary plus a
vowel offset: the pair with ko-kai is 7C99 where the consonant alone is 7C98,
the next vowel gives 7C9A, and so on to +5. Every consonant sits on a six-wide
block, itself plus a slot per leading vowel. And it is a contraction rather than
a swap, because the reverse order does NOT collide: consonant-then-vowel stays
two weights, 7C98 7C93.

Built as the rule rather than 220 transcribed entries, with each consonant's
primary read from the measured v0 table, so there is no hand-copied hex.

Also re-verifies the five DAO-only orders that are recorded as inert. They were
established over 31 samples of single characters, and French proved that shape
of evidence can hide an entire rule - it tailors no letter at all, so a word
with ONE accent looks identical to General and only two reveal it. Now 82
samples including words carrying two marks per script, and the Greek triple that
is the direct analogue of the French one. All five are still inert, and the
probe's positive controls still show their departures, so the null result means
the orders are inert rather than the harness being dead.

Every non-CJK sort order is now implemented: 28 of them, each verified against
ACE over 2,559 values.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI runs LibRed.Engine.Tests on five platforms - Linux, Windows, macOS and both
ARM legs - because LibRed is fully managed, and that run is what actually proves
the cross-platform claim. Twenty-four tests in there open an ACE OLE DB
connection, so they fail on four of the five, and for a reason that says nothing
about the code: the driver is simply absent.

They move to a new LibRed.Engine.AccessTests, which runs under the LibRedAccess
job beside LibRed.Core.Tests, where the other comparisons against the real
engine already live. They genuinely need the engine - QueryEngine, SQL, the lot -
so LibRed.Core.Tests could not host them without inverting the layering, and
LibRed.Ado.Tests would have meant filing engine tests under the ADO layer and
introducing an ACE dependency to a project that has none.

The five classes were already marked [Collection(AceCollection.Name)], so they
identified themselves and the split needed no judgement about which were which.

LibRed.Engine.Tests now has no System.Data.OleDb reference and no
AceTestDatabase, so this cannot drift back: an ACE test added there does not
compile rather than failing in CI on four platforms.

938 + 24 = 962, the count before the split.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
LibRed synthesises a new .accdb page by page rather than copying a packaged
empty file, so the collating order is a parameter of creation rather than a
property of a template. That had been demonstrated for the two General orders
and French; the other 27 configurations were an inference from sharing a code
path.

Now measured: 30 configurations - both General orders and every locale in
JetLocaleTailoring, including the two orders that exist at a second sort id
(German Phone Book, Hungarian Technical). For each, LibRed creates the file,
ACE creates a table and index INSIDE it, and every key ACE writes matches
LibRed's own. Nothing disagreed.

The bar is deliberately that rather than "the file opens". Two engines agreeing
on a shared index is the only check that catches a wrong key, because a
disagreement does not error - it makes seeks miss rows.

The list comes from asking IsIndexKeyEncodable rather than from a hardcoded set,
so it cannot drift out of step with JetLocaleTailoring and a new order is
covered the moment it lands.

It also guards the entanglement between the two: the system-table indexes are
built during creation, in the database's own order, so creating a database
REQUIRES encoding its collation. A locale with wrong weights would not merely
sort wrongly - it would make creation itself produce a file ACE disagrees with.
That circularity is why measuring a new locale for the first time needs DAO to
author the file.

Also corrects the CreateEmpty doc, which still said only the two General orders
could be encoded.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Access has two append forms. The single-record one takes VALUES and was already
supported; the multiple-record one takes a query and was not, which is the only
way to append more than one row in a statement - Jet has no multi-row VALUES
syntax at all, so "many rows" and "from a query" are the same feature there.

  INSERT INTO target [(field, ...)] SELECT [source.]field, ... FROM tableexpression

The IN externaldatabase clause both forms allow is deliberately left out:
appending into another file belongs to the linked-database subsystem LibRed does
not have, and a half-implementation would be worse than none.

Two behaviours were measured against ACE rather than reasoned, and one of them
caught a bug that all nine of my own tests had agreed with:

WITHOUT a column list, ACE resolves the source's output NAMES against the target
- not positionally, which is what this first implemented from the plausible
premise that only the count matters. The case that separates them is reversed
aliases: SELECT B AS Name, A AS Id emits ('seven', 7) in that order, and ACE
stores Id=7. Positionally it would have stored 'seven' in Id. ACE also rejects a
name the target lacks, SELECT * included, and LibRed now gives the same error.
WITH a column list the other rule applies: the list names the targets and values
map positionally onto it, whatever the source calls them.

Appending a table to ITSELF terminates. The source is materialised before a row
is written, or the scan consumes its own output forever; ACE doubles the table
and stops, and so does this.

The failure mode of getting the first one wrong is the silent kind - into
type-compatible columns it succeeds and puts the values in the wrong fields -
which is the argument for cross-checking rather than trusting green tests that
encode the same assumption as the code.

No cross-check exists for column DEFAULTs on the ACE side: its DDL rejects
DEFAULT in CREATE TABLE ("Syntax error in field definition"), a column property
Access sets through DAO/ADOX instead. That is a limit on what can be compared,
not a place the engines differ, and it is recorded in the test file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
  SELECT field1[, field2[, ...]] INTO newtable [IN externaldatabase] FROM source

A make-table looks like a SELECT and is an action query: it writes a table and
returns no rows. It runs with the writing statements, inside the implicit
transaction, so a failure part-way cannot leave a half-populated table.

Measured from ACE before implementing, because most of it could reasonably have
gone the other way:

  - A make-table copies DATA AND COLUMN DEFINITIONS ONLY. The source's PRIMARY
    KEY and its indexes are NOT copied, so archiving a keyed table gives an
    unkeyed copy.
  - A result column that IS a source column keeps that column's definition,
    width included: a source Text(30) arrives as Text(30). A COMPUTED column has
    no declared width to copy and gets Text at the 255 maximum.
  - An empty result still creates the table.
  - An existing target is an error - the docs call it "a trappable error" and
    ACE says "Table 'X' already exists".

The width rule cost a round trip worth recording. The probe output showed both
cases - a named column at Text(60) and a concatenation at Text(510), two lines
apart - and this generalised from the second while looking at the first. So the
cross-check against ACE caught a misreading of a measurement, not just an
unmeasured guess.

Two routing points the page cache and the tests found rather than review:
ExecuteQuery reaches ExecuteQueryCore without passing Route, so a make-table
invoked that way returned the source's rows; and Scoped classified any
SelectStatement as read-only, so the table write was refused by the guard that
rejects a write in a shared scope rather than silently upgrading it. Both now
account for INTO.

IN externaldatabase is not implemented, as on INSERT: creating a table in
another file belongs to the linked-database subsystem LibRed does not have.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MigrationsInfrastructureLibRedTest never completed: the class hung, and a
killed test host kept a handle on the .accdb so the next run could not
delete the file either. Diagnosed by sampling the stuck process — every
thread was parked in Thread.Sleep inside AcquireDatabaseLock — and then by
instrumenting the loop, which showed it spinning on a returned 0 and never
on an exception.

It was not a deadlock. The retry delay started at 1s and doubled while
under a minute, so after seven misses every contender retried once per 64
seconds; the delay never reset, and there was no jitter, so the threads
woke in lockstep and exactly one won per round. Fifteen threads therefore
took ~16 minutes, which is indistinguishable from a hang. Having no
timeout, the loop could not fail, only wait, so nothing was ever logged.

 - Retry policy: 50ms start, 1s cap, +/-25% jitter, and a one minute
   deadline that throws a TimeoutException naming the lock table and how
   to clear it. Build the lock object on success rather than allocating
   and discarding one on every attempt.
 - AcquireDatabaseLockAsync was missing both guards its synchronous twin
   has: no catch around the racy lock-table CREATE, none around the
   insert's duplicate-key race. Mirrored.

Fixing the wait exposed a second defect it had been hiding. Concurrent
migrators all pass the non-atomic exists-then-create check and all issue
CREATE TABLE; EF catches the losers as DbException, but LibRed threw
InvalidOperationException, which escaped that guard and failed the
migration outright. ACE raises OleDbException there, so translating is
what makes LibRed behave like the engine it stands in for.

 - New SchemaObjectExistsException, deriving from InvalidOperationException
   as ConstraintViolationException does, thrown from the four DDL name
   collisions: CREATE TABLE, CREATE VIEW/PROCEDURE, SELECT INTO and ALTER
   TABLE RENAME. LibRedCommand translates it into LibRedException with
   ObjectAlreadyExists (2714).
 - Assertions on those paths now name the exact type, since Assert.Throws
   does not accept a derived one.
 - New tests pin the lock contract that had none: the statement itself
   through the engine, the acquire/release cycle through ADO, and N
   connections contending, all bounded so a regression fails instead of
   hanging.

MigrationsInfrastructureLibRedTest goes from hanging to 34/36 in 84s. The
two remaining failures assert SQL Server baselines (sp_getapplock, CREATE
DATABASE, brackets) and were never ported. Engine 961/961, Ado 55/55, Core
789/789, Engine-ACE 32/32.

JetHistoryRepository is shared, so the ACE path gets the same retry policy
and async guards; that has not been exercised against a real driver.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
migrationBuilder.Sql("--Before") sends a command whose entire text is a
comment. The lexer skips WS/LINE_COMMENT/BLOCK_COMMENT outright, so that
text produces no tokens, and the statement rule has no empty production —
so it failed with "mismatched input '<EOF>' expecting {SELECT, IF, ...}".
Any user migration carrying a comment-only command hit this.

A comment is not a token, so this is not a missing statement kind and gets
no AST node: text with no tokens simply has no statement to run. The check
is answered by the lexer rather than by scanning for '--', because in
SELECT '--' the dashes belong to a string literal and a textual strip
would reduce a real statement to nothing.

 - ISqlParser.IsStatementless, implemented by pulling one token and asking
   whether it is already EOF.
 - QueryEngine.ExecuteQuery/Execute short-circuit to an empty result. They
   take no page scope: there is no work to isolate. Execute reports zero
   rows affected rather than a query's -1, a comment being an action that
   did nothing rather than a result set.
 - LibRedCommand.ExecuteBatch skips such a fragment rather than running it.
   ExecuteBatch returns the LAST statement's result, so without this
   "INSERT ...; -- done" would report the comment's zero rows in place of
   the insert's, and take @@rowcount with it.

Found in MigrationsInfrastructureLibRedTest, which died on the comment
before reaching anything it was testing. That test still fails, now on the
next command: its migration body is unported T-SQL (IF OBJECT_ID, THROW
65536, brackets), as is its baseline, and it fails on ACE too — it is
absent from the Jet green list.

Engine 970/970, Ado 59/59, Core 789/789, Engine-ACE 32/32.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reading MSysAccessStorage from the Northwind ACCDB failed outright:

  InvalidDataException: A relocation source must contain exactly one 4-byte
  pointer; found 55 bytes

RowRelocationReader required a relocation slot to be exactly 4 bytes wide.
That table has live overflow slots of 45-63 bytes, and the pointer is sitting
in the leading 4 bytes of each one.

The remainder is the row as it was BEFORE it moved, with only those 4 bytes
overwritten. Discount them and every field lands where the row format puts it
(the pointer covers the 2-byte column count plus the first 2 bytes of the
first column, leaving that column's remaining 6 bytes at offset 4); the
remnant's Id/ParentId/Type/Name equal those of the row it forwards to; the
remnant's null bitmap differs from its target's in exactly one bit, the OLE
column Lv, whose arrival grew the row and forced the move; and every remnant
is shorter than its target. The slot kept the old row's width instead of
being trimmed.

So the width was never the contract - the leading pointer is. Require at
least 4 bytes and read sourceBytes[..4]. The checks that actually validate a
relocation are untouched and unchanged: the target must be in the file, owned
by the same TDEF, and a nonempty hidden inline row.

What writes the wide form is NOT known, and is deliberately not claimed in
either the comment or the spec. Every writer reachable from code trims to
exactly 4 - measured across 317 relocations with no exception, covering ACE
on x64, the ACE 2010 runtime on x86, and LibRed's own writer, under growing
and shrinking text, repeated re-relocation, page fragmentation by interleaved
deletes, and an OLE column going from NULL to a value. Access's own
maintenance of its system tables is not reachable through SQL DML and remains
unexercised; the spec records that as the open avenue rather than guessing.

Because no write path produces the shape, the wide case is covered by handing
the resolver a wider source span directly rather than manufacturing it on
disk. A companion test pins our own writer still trimming, so if that ever
changes the spec's claim fails loudly instead of going stale quietly.

page-01-data-and-rows.md said "contains exactly one 4-byte pointer" and
"validates the exact source width". Both corrected, with the evidence.

Core 793/793, Engine 970/970, Ado 59/59, Engine-ACE 32/32.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JetByteArrayTypeMapping built its literal by appending "0x" and then each
byte as hex, so an empty array produced the bare string "0x". That is SQL
Server's empty-varbinary literal and it is not valid Access SQL: ACE answers
"Syntax error in query expression '0x'". The SQL was therefore wrong for both
providers - it failed on ACE exactly as it failed on LibRed, whose parser was
right to reject it (HEX_LITERAL already requires at least one digit, matching
ACE).

Verified against ACE, an empty STRING is the literal that works, in a value
position and in a DEFAULT: '' stores and reads back as a zero-length byte[],
and stays distinct from NULL (IS NULL does not match it). 0x00 is NOT the
same thing - that is a one-byte zero.

The parity half is the larger part. LibRed's codec cast straight to byte[],
so any string reaching a binary column threw InvalidCastException - not only
the empty one. Fixing just the empty case would have left 'A' broken.

ACE's rule, measured for VARBINARY and LONGBINARY alike, is that a string in
a binary column stores its UTF-16LE bytes:

    ''      -> byte[0]
    'A'     -> 4100
    'AB'    -> 41004200
    'e'     -> E900          (U+00E9)
    '41'    -> 34003100      the digits '4','1', NOT the byte 0x41

The empty case falls out of that rather than being special. JetTypeCodec now
routes Binary and Ole through AsBinary, which encodes a string that way and
passes a byte[] through untouched - the same UTF-16 treatment Memo text
already had.

Tests cover the rule, not just the fix: empty stores zero-length, empty is
not NULL, 0x00 is a one-byte zero, the four UTF-16 cases, hex still round
trips, and a digitless 0x is still rejected as ACE rejects it.

Spec check (LibRed.Core type-codec change): no docs/format update. The stored
representation is unchanged - raw bytes either way; what changed is which
input types the codec accepts. ACE treating binary as a Unicode string
throughout its SQL surface is worth documenting on its own terms once
explored (Len/LenB, comparison, concatenation), not as a footnote here.

Engine 979/979, Core 793/793, Ado 59/59, Engine-ACE 32/32.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A binary column has two faces in Access SQL, and LibRed implemented only one.
Measured against ACE:

    =, <, >, ORDER BY        BYTES - case-sensitive, byte order
    LIKE, Len, &, TypeName   a UTF-16 STRING, and LIKE carries the text
                             collation with it

so `B = 0x4100` matches only 'A', while `B LIKE 'A%'` matches 'A' (0x4100)
AND 'a' (0x6100) - the same column, compared case-sensitively by '=' and
case-insensitively by LIKE.

LibRed had the byte half exactly right already. The text half called
ToString() on the byte[], so it operated on the literal string
"System.Byte[]":

    Len(B)        13 for every value ("System.Byte[]".Length)   -> now 1
    B & 'x'       "System.Byte[]x"                              -> now "Ax"
    B LIKE 'A%'   0 rows                                        -> now 3
    TypeName(B)   "Byte[]"                                      -> now "String"

Silent nonsense rather than an error, which is worse than the
InvalidCastException fixed in d2264a0 on the write side.

The helper was already there: ToText() reinterprets a binary value as UTF-16
and was written for the byte functions (LenB, MidB, InStrB - which is why
LenB was correct all along). This just routes the text functions, both concat
operators, LIKE and TypeName through it as well.

TypeName now reports String for a binary column rather than the CLR type
LibRed actually holds. That is deliberate: ACE's expression service sees a
VT_BSTR there, and VarType already returned 8.

Verified end to end against ACE - Len/LenB 1/2 and 0/0 for empty, = 0x4100
matching one row, ORDER BY putting 0x4200 before 0x6100, LIKE 'A%'/'a%'/'A_'
matching 3/3/1, 0x0102 & 'x' giving "ȁx", TypeName String and VarType 8.
Every one of those is now a test.

Engine 989/989, Core 793/793, Ado 59/59, Engine-ACE 32/32.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ByteArrayTranslations.Any failed on both providers with "The LINQ expression
... .Any()' could not be translated", while its three siblings - Length,
First and Index - fail with the provider's deliberate refusal:

  Returning the exact length of a byte array is not supported by Jet ...
  There is support for a 'EF.Functions.ByteArrayLength' method ...

Those three are correct. LENB reports the UTF-16 byte count, so it rounds an
odd length UP to even and an exact length is genuinely unobtainable;
ByteArrayLength offers the LENB / ASCB(RIGHTB(x,1)) workaround and documents
precisely when it is wrong (data ending in 0x00, where a real even-length
value is indistinguishable from a zero-padded odd-length one). Refusing beats
returning a wrong number.

Any is different: it asks only whether there are any bytes at all, and that
question the rounding cannot spoil. An empty array is 0 and every non-empty
array is at least 2, so LENB(x) > 0 is exact - the trailing-0x00 ambiguity
cannot arise for a > 0 test, and unlike ByteArrayLength this needs no caveat.

Both baselines were still SQL Server's - DATALENGTH([b].[ByteArray]) > 0,
square brackets and all - i.e. copied and never ported, which is why nobody
noticed the translation was missing. Now:

  WHERE LENB(`b`.`ByteArray`) > 0

Verified on both providers, the Jet one against a real ACE driver: 9 tests,
6 passing where 5 passed before, and the same three by-design refusals.

A full functional run was not done; the new case is guarded on a parameterless
Any over byte[], so nothing else can reach it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ChrisJollyAU and others added 30 commits August 27, 2026 00:01
The DATETIME2 and BIGINT tests added over the last two commits assume the installed
ACE can create those columns. That holds on a developer machine running Microsoft
365, and does not hold on CI, which installs AccessDatabaseEngine_2016_x64 - Date/Time
Extended needs ACE 17 (Access 2019+), Large Number ACE 16. So they failed there for a
reason that says nothing about LibRed. My regression: written against the capability
in front of me without guarding for an older engine.

AceTestDatabase.SupportsColumnType asks the question once per type - CREATE TABLE with
that column on a throwaway copy - and caches it; the affected tests Assert.SkipUnless
on it with a reason naming the Access version required. No particular format version
is needed for the probe, since ACE raises the file itself when a column demands one.

It assumes an ACE that does not know a type name rejects it rather than silently
coercing it to something else. That is reasonable - ACE is strict enough about these
names to reject even DATETIME2(7) and DATETIMEEXTENDED as syntax errors - but it is an
assumption, so it is written down at the probe: if a guarded test ever fails on an
older engine instead of skipping, that is where to look.

Unverifiable locally, since this machine has the newer engine and correctly does not
skip (804 passed / 46 skipped, unchanged). The confirmation is CI turning 8 failures
into 8 skips.

Note this makes the DATETIME2 work unverified in CI rather than verified - green
because it skipped. Installing an ACE 2019+/365 redistributable on the runner, or
carrying both in the existing aceVersion matrix axis, would actually exercise it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…uest

The LibRedAccess job in pull_request.yml runs Core, Engine.AccessTests, Ado and
EFCore; push.yml was missing the Engine.AccessTests step, so those cross-checks
against the real Access engine only ran on pull requests and a push could go green
without them. Added verbatim from the pull_request version, comment included, so the
two job definitions now match step for step.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Introduced settings.json to explicitly allow only certain Bash and PowerShell commands (mainly dotnet and git). Added pre-tool-use hooks to deny shell commands that edit source files or run Python, and to restrict file reading commands to scratchpad/log files. This enhances safety and control over shell usage in the project.
The test for sysname column mapping to varchar(255) and nullability was removed from JetDatabaseModelFactoryTest.cs and LibRedDatabaseModelFactoryTest.cs. Related references were also deleted from ace_2010_odbc_x86.txt and ace_2010_oledb_x86.txt.
Both had drifted. AGENTS.md was the worse of the two, still claiming 10.0.x /
EF Core 10 / net10.0 and missing the OLE Automation heritage section entirely.

Corrections:

- LibRed is on master and builds as part of the solution; it was still
  described as living on a `libred` branch.
- JetMigrationsSqlGenerator does emit ALTER TABLE ... ALTER COLUMN. The claim
  that it cannot was flatly wrong.
- JetQueryTranslationPostprocessor now appends the identifier column to
  ORDER BY for deterministic tie-breaking, and reaches SelectExpression._identifier
  by reflection - worth knowing, since an EF bump breaks that at runtime.
- The test tree listed 5 projects; there are 12. Each is now tagged
  [Windows + ACE] or [cross-platform], which is what decides whether it can
  run at all. EFCore.Jet.Tests is called out as empty - its files are
  <Compile Remove>d, so a green run of it is not coverage.
- LibRed already does format version gating and auto-upgrade (JetVersion,
  AccessTypeMapper.RequiredVersion, JetDatabase.EnsureFormatAtLeast). BIGINT
  and DATETIME2 sit at different version bytes, 0x05 and 0x06, not the single
  ACE 16 threshold one might assume. The Jet provider has no equivalent and
  still maps long to decimal(20,0); that asymmetry is now stated so neither
  side gets "fixed" on an assumption about the other.

New sections: the green-tests baseline (the gate is "did anything that passed
stop passing", not the failure count), the CI job layout, the .docker images,
tools/, and the shell-edit restriction that .claude/settings.json enforces.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The EnableMillisecondsSupport option and all associated logic have been removed from Jet and LibRed providers, including interface properties, extension methods, internal state, configuration, and tests. Code handling millisecond support for DateTime, DateTimeOffset, and TimeSpan in Jet databases has been deleted. Migration test data for millisecond precision is now commented out, and test assertions use interpolated strings for expected SQL.
…m EFCore.Jet

Add a cross-platform EntityFrameworkCore.Jet.Common assembly holding the
Jet-dialect services shared by both providers, and move the bulk of
EFCore.Jet into it: query translation and SQL generation, migrations SQL
generation, conventions, update, value generation, logging and JetStrings.
Storage and Scaffolding stay in EFCore.Jet, which remains Windows-only.

Common keeps RootNamespace EntityFrameworkCore.Jet so the moved types and
the JetStrings resource name are unchanged.

The DUAL table name moves off JetConfiguration (it had no readers inside
EFCore.Jet.Data) onto the new JetDualTable, so the shared generator and the
scaffolding factory that detects the table can both reach it without the EF
layer depending on an ADO.NET driver. It stays user-settable via CustomName.

LibRed.EFCore now references only LibRed.Ado and Jet.Common. It registers
its own service graph rather than calling AddEntityFrameworkJet and patching
it, and gets 1:1 copies of the types that must differ per provider: options
and options extension, convention set builder, history repository, code
generator and design-time services. DataAccessProviderFactory is gone from
the LibRed test surface.

Move the [SupportedOSPlatform("windows")] stamp out of src/Directory.Build.props
onto the projects that need it, which lets src/LibRed/Directory.Build.props fold
back in with no overrides.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ests

LibRed's props set IsPackable=false and imported the repo-root props directly,
so the NuGet job's pack loop already ran over the five LibRed projects and
silently produced nothing. Import src/Directory.Build.props like every other
src project instead: they now pack with SourceLink, symbol packages and the
PackPdb switch, the same as the Jet packages. Only the documentation file is
turned back off. EFCore.Jet.Common needed no change - it already packed.

The NuGet job no longer depends on any other job. The code has been tested on
the pull request and again by this workflow's own test jobs, so making the
publish wait on a third pass only delays the package.

src/EFCore.Jet.Common was in neither path filter, so a Common-only change ran
no suite at all; it belongs to both, since both providers sit on it. The libred
filter's EFCore.Jet and EFCore.Jet.Data entries go: nothing under src/LibRed,
and no LibRed test project, references either one since the decoupling.
push.yml was also missing LibRed.Engine.AccessTests, which pull_request.yml
already had.

Drop the EF1001/CA1416 suppression from LibRed.EFCore - it was there for the
Jet internals the provider no longer extends.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t services

Introduce LibRedSqlMode (Extended = 0, the default; Compatible) and store it on
the LibRed options as SqlMode, replacing the earlier boolean. Extended is what
you get when nothing configures it; Compatible is opted into, either through the
new UseLibRed overloads that take the mode or via UseSqlMode on the builder,
whose parameter defaults to Compatible so calling it turns compatibility on.

Every existing UseLibRed overload gains a mode-carrying twin, so the set stays
symmetric: the no-connection, connection-string, connection and
connection-plus-ownership forms, each in the non-generic and <TContext>
families. On the connection path the three-argument form delegates to the
four-argument one with contextOwnsConnection false, mirroring how the existing
two-argument form already delegates.

Anything whose behaviour differs per mode now gets a 1:1 copy in LibRed.EFCore
rather than the shared type branching:

  LibRedQuerySqlGenerator          copy of the Jet generator; its factory picks
                                   between the two by SqlMode
  LibRedUpdateSqlGenerator         copy, still implementing Common's
                                   IJetUpdateSqlGenerator so Common's batching
                                   keeps working
  LibRedMigrationsSqlGenerator     copy
  LibRedQueryTranslationPostprocessor  copy that runs the ORDER BY lift only in
                                   Compatible mode; its factory injects
                                   ILibRedOptions to make that call

Each copy differs from its Jet original only in the namespace, the type name and
those gates - the bodies are otherwise byte-identical, so a diff against the
original stays readable.

With LibRed no longer using them, the Jet query translation postprocessor and
its factory move back out of Common into EFCore.Jet. Common keeps what both
providers still share on that path: JetLiftOrderByPostprocessor and
JetSkipTakePostprocessor.

The functional tests opt into Compatible explicitly, in ApplyConfiguration and
at the sites that build options directly, so they no longer depend on the
provider default. SqlMode is deliberately absent from the options LogFragment.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A copy of EFCore.LibRed.FunctionalTests that runs against LibRed's own SQL
generator instead of the Jet one. Extended is the provider default, so the copy
opts into nothing: every UseSqlMode call is removed, including the one in
ApplyConfiguration, and LoggingLibRedTest goes back to passing relationalAction
straight through.

Assembly name and namespace are EntityFrameworkCore.LibRed.Extended.FunctionalTests
so the two suites produce distinct fully-qualified test names - otherwise merged
reporting and stack traces could not tell them apart. Test discovery finds the
same number of tests as the compat suite.

A full copy rather than shared source or a shared base with two concrete
classes, for three reasons: the AssertSql baselines are inline in each test
method and nearly all of them will differ once the Jet workarounds are stripped
out; EF Core's own model is one project per target, each deriving from the
shared specification-test bases with its own baselines; and EF Core's baseline
rewriter replaces the whole AssertSql block, so anything cleverer than a plain
single baseline gets flattened by it.

The stores do not collide with the compat suite despite sharing names - each
project writes them under its own bin/<Config>/<TFM>, which is what already
keeps Jet and LibRed apart.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
LibRedQuerySqlGenerator no longer rewrites NULLIF(a, b) into
IIF(a = b, NULL, a). That rewrite exists because ACE has no NULLIF - it answers
"Undefined function 'NULLIF' in expression" - but LibRed's engine has
implemented NULLIF since a5aed06, with its own tests in
LibRed.Engine.Tests/NullIfTests.cs, so extended mode can emit the plain form.

The extended suite's five Conditional_uncoalesce baselines already expect
NULLIF, which is what they said before 0529e1c introduced the rewrite; they
arrived that way with the suite in fa01fe3 and have been failing since, because
the generator was still producing IIF. Removing the rewrite is what makes them
pass. The compat suite keeps IIF, since it still runs the Jet generator.

This is the first of the Jet workarounds to come out, and it was picked because
the engine already supported it: the two suites now produce different SQL for
the same tests off a single generator swap, which confirms SqlMode reaches
LibRedQuerySqlGeneratorFactory and that each suite selects the generator it
should. A failure from here on means the engine genuinely lacks something.

Also removes GreenTests/ace_2010_{odbc,oledb}_x86.txt from the LibRed
functional tests. The pass-lists belong to EFCore.Jet.FunctionalTests, whose
matrix is ACE version x architecture x ODBC/OLE DB; LibRed uses no driver, so
those legs do not apply to it and the files were strays.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
LibRedParameterBasedSqlProcessor is a 1:1 copy of the Jet one that runs two of
the passes in compatible mode only:

  JetOuterJoinProjectionGuardExpressionVisitor  wraps row-independent
      projections inside LEFT JOIN subqueries in a CASE the Jet generator needs
  JetCompatibilityExpressionVisitor             throws "Unsupported Jet
      expression" for ROW_NUMBER, CROSS/OUTER APPLY, EXCEPT, INTERSECT, JSON
      paths and some join predicates

Both are Jet-dialect workarounds. The second is the more consequential one to
drop: it is a rejection pass, so skipping it does not change any SQL text, it
stops extended mode refusing the constructs LibRed exists to support. Expect
those tests to move from a clean translation failure to whatever the parser,
binder, planner or executor says about them - that list is the capability gap,
and it is meant to be worked through gradually rather than all at once.

The other passes are unchanged and still unconditional: JetZeroLimitConverter,
JetDateTimeRangeConverter and JetSqlNullabilityProcessor.

Its factory injects ILibRedOptions to make the decision, the same way the query
translation postprocessor's does, and takes over the
IRelationalParameterBasedSqlProcessorFactory registration.

With LibRed no longer using them, the Jet processor and its factory move back
out of Common into EFCore.Jet, joining the query translation postprocessor pair
that moved earlier. Two doc comments in Common referenced the moved type with
<see cref>, which Common can no longer resolve; since CS1574 is suppressed
repo-wide that would have rotted silently, so they are now plain <c> text, which
is what the surrounding prose already does for types it does not link.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
27 test files, from the first extended-mode run: 36234 passing / 1567 failing /
624 skipped / 19 not run of 38444, against the same-day compat baseline of
36119 / 235 / 2071 / 19. The arithmetic balances exactly - 1447 = 1332 + 115 -
so every delta is a previously-skipped test that now runs, and nothing that
passed in compat regressed.

These baselines only exist for tests that pass: AssertSql runs after the query
has executed and its results compared, so a test failing on either never
reaches it and the rewriter never touches it. Anything still failing keeps its
original placeholder.

What changed, by count across the diff:

  412 [bracket] lines -> 0    upstream SQL Server placeholders in tests that had
                              never run, now real Jet SQL. The dominant category.
  FROM (  120 -> 17           nesting collapse from dropping the ORDER BY lift,
  ) AS    310 -> 136          which cost a derived-table layer per lifted
                              ordering. The largest structural change.
  10 interpolated $""" -> 0   baselines that had been hand-written in plausible
                              backtick SQL for tests that could never execute;
                              3 of them said CHARINDEX where the provider
                              actually emits LIKE.
  LEFT JOIN 77 -> 85          join predicates containing constants and
  INNER JOIN 74 -> 67         parameters now translate instead of being refused
                              by JetCompatibilityExpressionVisitor, whose
                              ContainsUnsupportCol rejected any predicate
                              holding a SqlConstantExpression.

CROSS APPLY, OUTER APPLY and ROW_NUMBER appear nowhere in the diff, on either
side - none of those reached a passing baseline, so they remain the bulk of the
1567 and are the next real capability gap.

Worth watching in the speed phase: without the lift, the correlated subquery is
written twice, once in the projection and once in ORDER BY, where the lift
materialised it once as a column.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…alue constructor

Four features, each one the plain SQL form that EF Core's base generator emits
once a Jet workaround is stripped out of the extended-mode generator. They share
the same files, so they land together. Engine suite 1104 passing, 0 failing.

ANSI paging - OFFSET n ROWS / FETCH FIRST m ROWS ONLY / both. Access has no
OFFSET at all, which is why the compat path emulates a page with nested TOP and
a reversed ORDER BY. The three shapes come from GenerateLimitOffset: a bare Skip
emits OFFSET, a bare Take emits FETCH FIRST, and together they emit FETCH NEXT.
FIRST and NEXT are synonyms in the standard, as are ROW and ROWS, so both
spellings parse either side; a grammar taking only NEXT would reject every plain
Take. Both operands reuse topOperand, so a parameter is accepted where Access
insists on a literal - EF passes the page bounds as @p, exactly what Jet's TOP
cannot take. LimitNode gained the offset and its count became optional (OFFSET
with no FETCH), and the sort bound is now offset + count, since the rows the
sort would otherwise discard are the ones the skip consumes.

Adding FIRST as a keyword broke the First() aggregate, caught by the suite. The
grammar already had the fix pattern - functionName readmits LEFT/RIGHT/ASC for
the same reason - so FIRST joins them. The other six new keywords were checked
against the function surface and the .sql files: no collisions.

CASE, both ANSI forms. Access has only IIF(), so the compat generator rewrites a
CASE into nested IIFs. The simple form is folded into the searched one at parse
time by rewriting each arm to `operand = value`, so evaluation sees one shape.
Arms are tested in order, only true selects (a NULL condition is skipped like a
false one), results short-circuit, and no match without ELSE is NULL.

The T-SQL reference turned up two things beyond the syntax. Aggregates inside a
CASE must be collected up front so they are computed per group - the standard
specifies the same order - and neither HasAggregate nor Aggregates could walk
into a CaseExpression, so HAVING CASE WHEN COUNT(*) > 1 failed with "Function
COUNT is not supported". Conditions matter as much as results there. And CASE
needed a declared type at all: the highest-precedence type across its branches,
where an untyped branch (a bare NULL) contributes nothing rather than erasing
the rest, numerics widen on the arithmetic ladder, and a genuine mismatch
declares nothing rather than guessing.

COALESCE - the first argument that is not NULL, NULL when all of them are. The
standard makes it shorthand for a CASE over its arguments, so it takes the same
type rule and reuses that helper. SQL Server implements the shorthand literally,
which is why its own docs warn arguments are evaluated more than once and a
subquery argument can differ between evaluations; each argument is evaluated at
most once here, same answer without the instability.

NULLIF's semantics were already right and already tested, but it had no declared
type. It returns its first expression or a NULL of that expression's type, so
unlike COALESCE it takes the first argument's type outright - the second only
takes part in the comparison.

The table value constructor, both of its uses. As an INSERT's VALUES clause it
now takes a list of rows rather than one; the AST and executor were already
row-list shaped, so only the grammar and builder needed it. DEFAULT is allowed
as a row value, carried through as a sentinel rather than a value because it has
to be distinguishable from NULL: on a column with a default, NULL stores NULL
while DEFAULT takes the default. A column with no default stores NULL, and a
required column with no default is refused rather than written as NULL. As a
query - VALUES (1), (2) as an operand of a set operation - it is the shape EF
emits for an inline collection, and its expressions may reference outer columns,
so they are evaluated per outer row rather than folded once. Column names come
from the leading query, per SQL, so no column alias list is needed; EF does not
emit one, checked against the failure set.

DEFAULT is confined to the INSERT clause, which the standard requires and the
grammar gets for free by admitting rowValue nowhere else. A stored append
procedure keeps its values as text, so it rejects both a multi-row constructor
and a DEFAULT value - there is nothing to store for either.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Extended mode now emits the plain SQL form in the places the engine learned to
handle in the previous commit, so the Jet-dialect rewrites come out of
LibRedQuerySqlGenerator: the CASE-to-nested-IIF expansion, the COALESCE
rewrite, and the surrounding null-guard scaffolding they needed. The compat
generator in Common keeps every one of them - Access has no CASE, no COALESCE
and no OFFSET, so that path still needs all of it.

The postprocessor also stops running JetSkipTakePostprocessor in extended mode.
That was the double-TOP-and-reverse emulation of a page, which existed only
because Jet has TOP and nothing else; with native OFFSET/FETCH in the engine,
EF's own paging form goes straight through. It joins the ORDER BY lift as the
second pass gated to compat.

Together these are what turn a query the generator used to rewrite into one it
emits verbatim - which is the whole point of extended mode, and why the engine
work had to land first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
71 files, from the run that took extended-mode failures from 16185 to 1696
against a compat baseline of 235. The arc, each step a single feature:

  16185  after gating skip/take to compat, before the engine could page
   5462  native ANSI paging          - cleared ~10,700, the largest jump so far
   2549  standard CASE               - cleared ~2,900
   1696  COALESCE and the table value constructor - cleared ~850

Paging dominates because 13844 of those 16185 failures carried "FETCH" in the
error message: one missing grammar rule, not 13844 problems. Worth remembering
as a triage habit - group the failures by error text before counting tests.

The baselines only exist for tests that pass, since AssertSql runs after the
query has executed and its results compared; anything still failing keeps its
original placeholder. So these diffs are the newly-working queries, and the SQL
in them is markedly plainer than the compat suite's: no nested TOP for a page,
no IIF chain for a CASE, and far fewer derived-table layers now the ORDER BY
lift is gone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Access has no CROSS JOIN keyword - a cartesian product is written there as
comma-separated sources in the FROM clause - so the grammar had only the comma
form, and joinClause made ON mandatory, leaving no shape for a join without one.
EF Core's base generator emits the explicit keyword, so LibRed now takes both
spellings for the same thing.

Small change, because JoinKind.Cross was already plumbed through for the comma
form: the planner, executor and index selection all handle a Cross join with a
null condition already. CROSS JOIN builds the identical JoinTable node, so only
the syntax was missing.

joinClause splits into labelled alternatives to keep ON mandatory for the
conditional kinds and absent for CROSS. That made JoinClauseContext abstract and
broke the stored-view decomposition, which turned out to fit rather than fight:
Access records a view's joins by their ON condition (Name1/Name2), so a CROSS
JOIN contributes a source and no join entry - exactly how the comma form was
already stored. The two spellings decompose identically, so nothing needed
rejecting.

CROSS was checked for identifier collisions before being made a keyword: no bare
occurrence in any .sql file, and it is not a function or aggregate name.

The tests pin the equivalence directly - CROSS JOIN and the comma form return
byte-identical result sets - along with ON being rejected on a cross join and
still required on a conditional one, so the new alternative did not loosen the
existing rule.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The largest strip so far: a whole copy of the base VisitSelect, carrying five
unrelated Jet behaviours at once. Extended mode now uses the base method, so the
FROM clause is generated by GenerateFrom like every other provider.

What went with it, and what each was for:

  parenthesised join nesting   Jet is fussy about which join may follow which, so
                               VisitJetTables grouped them: FROM ((a JOIN b) JOIN c)
  the DUAL pseudo-FROM         a FROM-less SELECT needed a one-row table to select from
  empty-projection substitution  SELECT 1, or a column lifted out of the WHERE
  the colexp IS NOT NULL guards  appended to WHERE to restore inner semantics after
                               a join had been forced outer
  comma joins                  Access's only cartesian product spelling

Four of those landed with no breakage at all. The entire cost was the fifth: base
GenerateFrom emits an explicit CROSS JOIN, which the grammar did not accept until
the previous commit. Failures went 1696 -> 2137 on the removal and 2137 -> 1693
once CROSS JOIN landed, so net -3 - but the count understates it, because the SQL
is markedly closer to standard. 1476 fewer parenthesised join groups, 355 LEFT
JOINs back to INNER, 308 fewer IS NOT NULL guards, 255 explicit CROSS JOINs.

The LEFT-to-INNER change is worth understanding: those joins were INNER to begin
with. EFCore.Jet manufactured a LEFT plus a null-rejecting WHERE guard to satisfy
Jet's join-ordering rules; nothing is being optimised now, the fabrication has
simply stopped. LibRed has no such rule - tableSource is a flat chain with a kind
per node, so a join following a join is just a join.

GenerateTop and GenerateLimitOffset stay. They are separate overrides and match
upstream SQL Server's exactly, which likewise has no VisitSelect override: TOP
for a bare limit, OFFSET/FETCH NEXT once there is a skip. TOP is not a Jet
workaround - it is native to the dialect LibRed implements, and the engine has
supported it from the start - so seeing it survive in the baselines is correct
rather than an oversight.

VisitJetTables remains for now, called only from VisitDelete and VisitUpdate,
which still emit Jet-shaped joins. The rebaseline confirms the split is clean:
zero DELETE or UPDATE lines moved across 69 files. Those two are a separate
change, after which the helper goes entirely.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
69 files. Every change is in the FROM clause - not one CASE, COALESCE, NULLIF,
#Dual, bracket placeholder or UNION line moved, and TOP/OFFSET/FETCH each shifted
by a single incidental line. Counted across the whole diff:

  FROM (        1676 -> 200    Jet's join parenthesisation, not derived tables:
                               `) AS` is 626 on both sides, so derived tables are
                               untouched. FROM ((a JOIN b) JOIN c) -> FROM a JOIN b JOIN c
  INNER JOIN     618 -> 973    +355, exactly offsetting the LEFT JOIN fall
  LEFT JOIN     2327 -> 1972   -355
  IS NOT NULL    347 -> 39     -308, the guards that accompanied those joins
  CROSS JOIN       0 -> 255    with comma-form FROM lines 127 -> 0

The three middle rows are one edit, not three. EFCore.Jet turned an INNER JOIN
into a LEFT JOIN plus a null-rejecting WHERE guard because Jet restricts which
join may follow which; base GenerateFrom simply emits the INNER JOIN that was
there all along.

DELETE and UPDATE are absent from the diff entirely - zero lines either side, and
RIGHT JOIN unchanged at 9 - because VisitDelete and VisitUpdate still route
through VisitJetTables. That path is a separate change and these baselines will
not move again until it lands.

Failures 1693, from 1696 before the strip.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lateral joins: the right side is evaluated once per left row with that row
in scope, so it may correlate to the left. ACE has neither keyword, but EF
Core's base generator emits both, so extended mode needs them.

Grammar follows T-SQL's positioning, which lists
`left_table_source { CROSS | OUTER } APPLY right_table_source` as a
<joined_table> alternative next to the conditional joins and CROSS JOIN:
two more joinClause alternatives taking no ON, since the correlation inside
the right side is the condition.

Represented as two new JoinKind members rather than a separate node type, so
every kind-agnostic walker keeps working untouched and every optimizer site
that switches on kind - predicate pushdown, sort pushdown, index-nested-loop
and hash-join conversion - declines a lateral join by construction, because
each is an allowlist. Execution needs its own path regardless: ExecuteJoin
materialises the right side once against the enclosing scope, while
ExecuteApply re-runs it per left row with that row pushed onto the scope
chain, the mechanism a correlated subquery already uses. The joined schema
has to be known before any left row is read, so the right side is probed once
for its columns.

Two planner gaps surfaced with it, both of which made a nested APPLY
(Select_nested_collection_deep) look like a hang at ~62.7M rows:

- Place never descended into an APPLY. Both kinds preserve the left, so a
  conjunct confined to it can be pushed there. Nothing is pushed into the
  right side: under OUTER APPLY a filter there can empty an otherwise
  non-empty result and manufacture the null-padded row the WHERE was
  dropping.
- A conjunct naming an OUTER alias was never a pushdown candidate, because
  Place demanded every qualifier be inside the subtree. PushPredicates now
  takes the aliases the FROM introduces and ignores the rest, an outer column
  being readable at any depth. General rather than APPLY-specific; it stayed
  invisible while a correlated predicate still landed as a Filter directly
  over its scan, which RewriteFilterOverScan already catches.

Index selection additionally plans a lateral right side with the left's
aliases as visible outer aliases, so a correlated predicate there becomes a
seek instead of a rescan per left row.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The AssertSql baselines these tests carried were SQL Server's, never ported,
because JetCompatibilityExpressionVisitor rejected APPLY and the test utility
turned that into a dynamic skip. With the visitor gated to compat mode the
queries run, and the rewriter has now written what LibRed actually emits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CVar(NULL) on the producing side and IIF(x IS NULL, NULL, CLNG(x)) on the
consuming side were one workaround in two halves: Access loses the type of a
bare NULL literal in a projection, so it was tagged on the way out and
re-coerced on the way back in. LibRed carries declared types through the plan,
so both halves go together - removing either alone would have been wrong.

TryGenerateWithoutWrappingSelect no longer peeks a parent stack for an ambient
InExpression; it tests directly for a select whose only table is a
ValuesExpression, which is the condition it actually wanted. That was the
stack's last live reader, so the stack, its five push/pop pairs, the GenerateIn
override and _nullNumerics all go with it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CVar(NULL) becomes NULL and the IIF/CLNG guard around the column reading it
disappears. The bulk is TPC inheritance, which unions sibling tables and pads
each absent column with NULL, so the wrapper appeared once per padded column
per union arm. Also picks up the VALUES term now wrapped in its own subquery.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
EF Core's base generator emits ROW_NUMBER() OVER (PARTITION BY … ORDER BY …)
and the parser rejected it outright, which is the largest single block of
failures left in the extended suite. Access has no window functions at all, so
this is a LibRed extension.

Built so that adding another one costs a registry entry and nothing else. OVER
hangs off `functionCall` rather than off a list of window-function names, so
RANK, NTILE and friends already reach it through `functionName` at no grammar
cost, and `SUM(x) OVER (…)` parses for free. RANK and DENSE_RANK are included
to show that: four lines each, no grammar, no regeneration, no AST, no planner,
no executor.

PARTITION had to become a keyword for PARTITION BY, but Access has a real VBA
Partition(number, start, stop, interval) that LibRed implements and tests, so
it is readmitted as a function name beside FIRST - a call is always followed by
'(' and PARTITION BY never is. OVER is a plain new reserved word.

WindowFunction is deliberately NOT a FunctionCall subtype: HasAggregate matches
any FunctionCall whose name is an aggregate, so a windowed aggregate would make
the query look grouped and build a bogus AggregateNode.

The planner lifts each call out of the projection into a reference to a
synthetic column WindowNode publishes, so DeclaredType, ProjectionSchemaFor,
sorting, DISTINCT and LIMIT all keep seeing an ordinary column and needed no
changes. WindowNode preserves input order - the sort sits above it, so emitting
in partition order would silently reorder any windowed query with an ORDER BY.

Two optimizer sites needed care rather than luck. IndexSelection.Apply's
default arm stops descending, not just rewriting, so without a WindowNode case
every windowed query would silently become a full scan - correct results,
catastrophic plans, the failure mode that made nested APPLY look like a hang; a
plan-shape test guards it. SubtreeAliases needed the same pass-through case
ProjectNode's comment records having needed. PushPredicates needs nothing: the
WindowNode is built after pushdown has run.

A window over a grouped query throws rather than silently misbehaving:
AggregateNode owns the projection and collapses rows, and nothing EF emits
needs it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Window functions now execute, so the ~180 tests that emitted
ROW_NUMBER() OVER(PARTITION BY … ORDER BY …) run for the first time and their
baselines are written from LibRed output rather than the SQL Server text the
project inherited when it was copied.

Multiple_members_of_correlated_single_result_subquery_lift_to_single_join had
its `method switch` flattened by the rewriter, which can only write one
baseline per test - so the First/Single arm was kept and the Last and ElementAt
arms were lost, leaving 8 of its 16 cases failing. The switch is restored with
each arm's real output: Last orders the window DESC, ElementAt uses the
two-sided 0 < row AND row <= 1 predicate. It is the only test in either suite
whose AssertSql sits inside a switch, so the exposure is that one test.

Checked before committing: nothing a LibRed run cannot produce appears on the
added side - no SQL Server TOP(n), no N'' literals, no @__ parameter names, and
the single bracketed line is raw SQL the keyless-entity fixture supplies as
input. All 9 removed APPLY baselines are stale (6 SQL Server, 3 Jet-era).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The hook payload carries "scratchpad_dir" on every call, and the exemption
(scratchpad|\.log|\$log|/tmp/) was grepped against the whole payload - so it
matched unconditionally and the read guard exempted itself on every command.
It has never fired since it was written. Checks now run against the payload
with the envelope fields stripped (scratchpad_dir, transcript_path, cwd, and
description, so model-written prose cannot trigger or exempt a rule either).

The two hooks are merged into one that reads stdin once. They both consumed
stdin independently, which happened to work but meant a third check could
break the second; one read removes the ordering dependency.

A deny-only hook leaves everything else to the permission system, so a
legitimate scratchpad read still cost a prompt. The exempt case now returns an
explicit allow. Because that grants rather than blocks and the exemption is a
text match, the allow branch declines when the command contains ; & ` or $( ,
so a chained command merely mentioning scratchpad falls through to a prompt
instead of being auto-approved. Plain pipes still qualify.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`x IN (SELECT … UNION ALL SELECT …)` failed to parse: IN, EXISTS and a scalar
subquery each took a bare selectStatement, while a derived table has always
taken a full queryExpression. The standard reaches all four positions through
the same <query expression> nonterminal, so this was an internal inconsistency
rather than a deliberate narrowing.

EF Core emits the flattened form once its generator elides the wrapping select
it would otherwise put around the union - which 51a2b24 enabled by removing
the parent-stack peek that had been forcing that wrapper inside an IN. The
generator is right and the engine was behind.

The three AST records now carry SqlStatement, matching SubqueryTable, rather
than the parse being desugared into a synthetic derived table. That makes the
compiler enumerate every consumer instead of leaving them to be found by
inspection - it turned up AggregatesInSelect and ViewExpander, neither of which
was on the list drawn up beforehand.

Consumers split two ways. Those that only run the body take it as it comes:
SubqueryPlan now calls PlanStatement rather than PlanSelect. Those that inspect
its shape - the EXISTS, IN and scalar-aggregate decorrelation rewrites - read a
projection, FROM and WHERE that a set operation does not have, so each call
site now declines anything that is not a plain SELECT. Declining costs speed,
never correctness: the per-row path runs the body as written. AggregatesInSelect
is the exception and walks both arms, because an aggregate over an outer column
left uncollected would fail to resolve rather than merely run slower.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`SELECT … UNION SELECT … ORDER BY x` returned the rows in the wrong order and
said nothing: ORDER BY and OFFSET/FETCH lived on selectStatement, so a trailing
one bound to the LAST OPERAND. Measured against ACE with the probe added here -
over UA(50,10) and UB(40,20), ACE returns 10,20,40,50 where LibRed returned
50,10,20,40, which is UA unsorted followed by UB sorted.

The grammar now states where the clauses may appear rather than a builder
correcting it afterwards. selectStatement becomes an order-less
querySpecification - the standard'"'"'s own name for that rule, and Trino'"'"'s - and
queryExpression carries `orderByClause? offsetFetchClause?`, which is the
standard'"'"'s structure: <query expression> is ordered, <query term> is not. Every
serious SQL grammar factors it this way (Trino, SQLite'"'"'s select_core,
PostgreSQL'"'"'s simple_select) for the reason met here, that an operand rule ending
in an optional ORDER BY swallows the enclosing expression'"'"'s greedily.

That also removes an ambiguity rather than working around it: SelectStatement.Top
holds either a leading TOP or a trailing FETCH, and lifting the right one used to
need a peek at the parse tree. Now a leading TOP is on the operand'"'"'s
specification and a trailing FETCH on the expression, and they cannot be
confused.

SetOperationStatement carries the clauses; the planner puts SortNode/LimitNode
above the set operation. A single term folds them back onto its SELECT, so an
ordinary `SELECT … ORDER BY x` builds exactly the AST it always did - pinned by
a test, since that is the case that would ripple everywhere if it changed.

Two consequences worth knowing. An operand may still be ordered by
parenthesising it, and must be: `(SELECT TOP 5 … ORDER BY x) UNION …` is a
nested query expression, and the ordering is what makes that TOP deterministic.
And ORDER BY on a non-final operand is now a parse error where ACE parses and
ignores it - a deliberate divergence, since ACE ignoring it means no such query
ever depended on it, and refusing loudly beats accepting silently.
EF emits a correlated COLUMN as the skip for `ElementAt(<column>)`:
`OFFSET `s`.`Id` ROWS FETCH NEXT 1 ROWS ONLY`. offsetFetchClause reused
topOperand, which admits only a literal, a parameter or a parenthesised
expression, so that failed to parse.

The two rules are no longer shared, because the restriction was never about
paging. TOP has to stay narrow: it sits immediately before the select list,
where an unrestricted expression would swallow the star of `SELECT TOP 5 * FROM
t` - the grammar already said so. A paging count is closed by the ROW/ROWS
keyword that must follow it, so there is nothing there to swallow and it can be
a full expression.

Wider than what is written down, and deliberately so. The standard'"'"'s <offset row
count> is a <simple value specification> - literal, parameter or variable - and
SQL Server documents offset_row_count_expression as a variable, parameter or
constant scalar subquery; a correlated column is none of those. But SQL Server
accepts one, which EF Core'"'"'s own SQL Server baseline for
Where_subquery_with_ElementAt_using_column_as_index demonstrates: it carries
`OFFSET [s].[Id] ROWS`, and exists only because that test passed against a real
server. This matches the engine, not the docs.

Nothing was needed in the executor: LimitNode already evaluates its bounds
against the enclosing scope, which is where a correlated column resolves - and
only there, since the bounds are evaluated with an empty row schema, so a
reference to the paged query'"'"'s own columns cannot resolve at all.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four files move. The GearsOfWar trio picks up `x IN (<union>)` now that a
subquery predicate takes a full query expression, so the wrapping select EF used
to be forced to emit is gone from the baselines.

PrimitiveCollections picks up the correlated-column OFFSET, and is worth a look
as the case where several of these compose: a set operation mixing a SELECT with
a VALUES term, used as a correlated derived table, with an outer column inside a
row value, ordered, and paged by `OFFSET `p`.`Int` ROWS`. It also shows the
column-alias-list decision holding - SQL Server writes `AS [v]([_ord], [Value])`
where EF names the columns from the leading SELECT of the union for us, which is
exactly why that list was never needed.

Checked before committing: nothing a LibRed run cannot produce appears on the
added side - no SQL Server brackets, TOP(n), N'' literals or @__ parameters -
and the OFFSET and UNION lines are replaced one for one rather than lost.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant